Neural Network Models of Adaptive Behavior

The paper characterizes the field of investigations “Simulation of Adaptive Behavior” that is on the border of biology and informatics and studies architectures and informational processes of control systems that govern behavior of both animal and artificial creatures (animats). The project “Animat Brain” (a neural-network-based framework for simulation of adaptive behavior) is described. 1 Simulation of Adaptive Behavior In the early 1990s, the animat approach to investigation of adaptive behavior was proposed [1]. The term “animat” originates from two words: animal + robot = animat. This research methodology implies understanding animal intelligent features through simulation of artificial animals ("animats") in progressively more challenging environments. According to the AnimatLab [2], the goals of the animat approach are as follows: "On the short term, the objectives of the animat approach are first to understand the mechanisms that afford animals the possibility of adapting and surviving, and then to import such mechanisms within artifacts “e.g., autonomous robots” capable of adapting themselves and of fulfilling their mission within more or less changing and unpredictable environments... On the long term, the objective of the animat approach is to contribute to the advancement of cognitive sciences through the study of how human intelligence is rooted in simpler adaptive behaviors inherited from animals, in a bottom-up, evolutionary and situated perspective." Investigations of adaptive behavior are based on serious computational methods: Neural networks Genetic algorithm [3] and other methods of evolutionary computations Classifier Systems [4] Reinforcement Learning [5]. The next section outlines the project “Animat Brain” that is intended to develop a rather general framework for modeling animat behavior. 2 The project “Animat Brain” The project “Animat Brain” is based on neurophysiological theory of functional systems [6]. Functional systems were put forward by Petr K. Anokhin in the 1930s as an alternative to the predominant concept of reflexes. Contrary to reflexes, the endpoints of functional systems are not actions themselves but adaptive results of these actions. Initiation of each behavior is preceded by the stage of afferent synthesis. It involves integration of neural information from a) dominant motivation (e.g., hunger), b) environment (including contextual and conditioned stimuli), and c) memory (including innate knowledge and individual experience). The afferent synthesis can occupy a substantial time and involve cycles of reverberation of signals among various distributed neural elements. The afferent synthesis ends with decision making, which results in selection of a particular action. A specific neural module, acceptor of the result of action, is being formed before the action itself. The acceptor stores an anticipatory model of the required result of a goal-directed action. Such model is based on a distributed neural assembly that includes various parameters (i.e., proprioreceptive, visual, auditory, olfactory) of the expected result. Performance of every action is accompanied by a backward afferentation. If parameters of the actual result are different from the predicted parameters stored in the acceptor of action result, a new afferent synthesis is initiated. In this case, all operations of the functional system are repeated until the final desired result is achieved. The current version of the “Animat Brain” develops the previous version of the project [7] and uses also the approaches to modeling intelligent adaptive behavior proposed in the project “Animal” [8], as well as the approaches of the Neuroscience Institute [9]. It is supposed that the animat control system consists of neural network blocks and is analogous to animal control system. Each block is a formal functional system (FS). The highest level of the control system hierarchy corresponds to survival of the simulated animal. The next level corresponds to the main animal needs (energy replenishment, reproduction, security, knowledge acquisition). Blocks of lower levels correspond to tactical goals and sub-goals of behavior and have no rigid hierarchy. At any time moment, only one FS is active, in which the current action is formed. Proceedings of the 1 International Conference on Mathematical Biology and Bioinformatics, Pushchino, Russia, 2006 The FS described here reflects the following important features of its biological prototype: a) prognosis of the action result, b) comparison of the prognosis and the result, and c) correction of prognosis mechanism via learning in appropriate neural networks. Each FS consists of two neural networks: the Controller and the Model. At any time moment t (t = 1,2,...), the operation of the active FS can be described as follows. The state vector S(t) characterizing the current external and internal environment is fed to the FS input. The Controller forms the action A(t) in accordance with given state S(t), i.e. the Controller forms the mapping S(t) -> A(t). Some actions A(t) are commands onto effectors (actual actions), another actions are activation commands for other FSs. The Model predicts the next state for given vectors S(t) and A(t), i.e. the Model forms the mapping {S(t), A(t)} -> Spr(t+1). The mappings S(t) -> A(t) and {S(t), A(t)} -> Spr(t+1) are stored in synaptic weights of neural networks of the Controller and the Model. Each of these networks has feed-forward structure. It is supposed that the animat is trained without a teacher, as a result of direct interaction with external environment. The animat receives positive or negative reinforcements that are related with the basic animat needs. For example, if the need of energy replenishment is satisfied then the positive reinforcement is received. The learning of neural networks is carried out as follows. There are two regime of learning: 1) the extraordinary mode and 2) the fine tuning mode. The extraordinary mode is a rough search of behavior that is adequate to the current situation. This mode comes, if there is a strong negative reinforcement corresponding to some main need (strong pain or strong feeling of famine), or there is something quite unexpected (the state prediction Spr(t+1) in the active FS strongly differs from the real state S(t+1) ). In the extraordinary mode, a random search for new behaviors takes place; namely, new functional systems are randomly generated and selected. In the case of a strong negative reinforcement, the selection is intended to search for FSs that execute adequate actions. In the case of an unexpected situation, the selection is intended to search for FSs that predict future states correctly. This mode is similar to neural group selection in the theory of Neural Darwinism [10]. In the fine tuning mode, learning is adjustment of synaptic weights of Controllers neural networks in the FS that is active at the current moment of time and in the FSs that were active in some previous steps of time. At positive/negative current reinforcement, the relations between inputs and outputs in neural networks of Controllers are amplified/relaxed. As synaptic weights are updated in those blocks, which were active in previous time steps, this learning mode allows forming chains of consecutive actions. The synaptic weights of Model neural networks are adjusted in order to minimize the prediction errors, error = ||Spr(t+1) S(t+1)||. It should be underlined that though described scheme of animat control system implies definite block architecture that is convenient for its modeling, the proposed mechanisms of learning allow automatic allocation of blocks in distributed neural networks.